CAPSTONE PROJECT - THE BATTLE OF NEIGHBORHOODS

VISITING LONDON CITY

Applied Data Science Capstone by IBM/Coursera. Author: Jorge Alvarez

Table of contents

• Introduction
• Business Problem
• Data
• Methodology
• Explanatory Analysis
• Results and Discussion
• Conclusion

1. Introduction

London is the capital and largest city of England and the United Kingdom. Its estimated mid-2018 municipal population (corresponding to Greater London) was roughly 9 million, which made it the third-most populous city in Europe. London accounts for 13.4% of the U.K. population. Greater London Built-up Area is the fourth-most populous in Europe with 9,787,426 inhabitants at the 2011 census.

London has been one of the world's top tourism cities for many years, and a key gateway for domestic and international visitors. The total number of visitors in London, including domestic tourists, is estimated at 12 million per year.

As a result, tourism is the second most important sector for the economy of the city after financial services, contributing 12% to its GDP. Tourism, therefore, plays a vital role for London. The sector employs 700,000 people and contributes £36 billion a year to the economy. Tourism also helps to strengthen London's reputation as an open and welcoming global city.

2. Business Problem

London is a world tourist destination with many visitors each year. The information available concerning tourism in London can be overwhelming and considering trips made to London for holiday purposes lasted 4.8 nights on average in 2019, according to Statista, a good planning is necessary to make the most of the holidays, avoiding downtimes. London has full potential to become the best city to visit in Europe but it still has challenges for visitors such as not using public transport wisely, waiting in long queues as a result of a poor planning and failing to research available attractions.

This project will analyse the venues available for people visiting London in order to look for the best recreational activities and locations in the variety of districts in London city. The venues will be analyzed first as a whole and structured afterwards by district for a better comprehension as well.

This project will be interesting to visitors and expats who are considering visiting and relocating to London as they will be able to identify the most attractive locations in London and explore its districts and common venues around each district.

3. Data

This section describes the data that will be used to solve the problem concerning the tourism and visits to London city.

3.1. Libraries

The libraries installed and imported to be used in this project are summarized below:

• Numpy: Library to handle data in a vectorized manner. It is used to create cluster labels during the clustering stage.

• Pandas: Library for data analysis. In this project is used to build DataFrames containing Postal Codes, districts, venues, latitudes, longitudes.

• Requests: Library to handle requests. It is used to scrape the list of Postal Codes in London city from Milesfaster website (please, refer to sources section below).

• BeautifulSoup: Library used for pulling data out of html and XML files. It is necessary to scrape information from web pages.

• Geopy: Library used to convert an address into latitude and latitude values. It is used to add the latitudes and longitudes to the districts scraped.

• Json: Library to handle JSON files. In this project is used to make the GET request during the neighborhood exploration stage.

• folium: Library used to create maps. It is used to create maps in London city, containing all districts previously scraped.

• matplotlib.pyplot: Library used to plot bar plots.

• Matplotlib.cm and Matplotlib.colors: Library used to plot a range of colors in London city maps.

• WordCloud: Library used to plot a world cloud containing the most common venues in each district.

• k-means: Library used to apply k-means clustering algorithm used to cluster districts in London city.

3.2. Data Acquisition

The data acquired for this project contains the following columns:

• Postal Code: Contains the postal codes of London city that have the following coding. C: Central, N: North, E: East, W: West, SE: South East, SW: South West, NE: North East, NW: North West.

• District: London city districts within a Postal Code.

• Postal Code Latitude: Latitude of a given Postal Code in London city.

• Postal Code Longitude: Longitude of a given Postal Code in London city.

• Venue: Venue names within a district in London city.

• Venue Latitude: Latitude of a given venue in London city.

• Venue Longitude: Longitude of a given venue in London city.

• Venue Category: Category of a given venue (such as bakery, coffee shop, movie theater, etc.)

• Most common venue: The most common venue within a district ranging from 1st to 10th.

3.3. Data sources

For this project the following data and sources will be used:

• Milesfaster website This is a London Hotels specialist website which contains the postal codes and districts of London city. The postal codes and district data of London city will be scraped from Milesfaster using the beautiful soup library of Python, however, it requires an additional data cleaning step to reorganize and label columns. Milesfaster url is the following: https://www.milesfaster.co.uk/london-postcodes-list.htm

• ArcGIS Geocoding is the computational process of transforming a physical address description to a location on the Earth's surface (spatial representation in numerical coordinates). ArcGIS will be used to turn addresses into coordinates, finding addresses interactively and geocoding a table of addresses.

• Foursquare API to get the most common venues of given district in London city. Foursquare API has one of the largest database of 105+ million places and is used by over 1250,000 developers. It provides different data information of venues among districts. Foursquare data contains venues, longitude, latitude and postal codes. The information obtained per venue as follows: district, district latitude, district longitude, venue name, venue latitude, venue longitude, venue category. This will be used in the methodology section.

Installing and importing libraries

Before exploring the data, the necessary libraries will be installed and imported.

import numpy as np # library to handle data in a vectorized manner

import pandas as pd # library for data analysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import json # library to handle JSON files

!conda install -c conda-forge geopy --yes
from geopy.geocoders import Nominatim # convert an address into latitude and longitude values

# install and import geocoder
!pip -q install geocoder
import geocoder

import requests # library to handle requests
from pandas.io.json import json_normalize # transform JSON file into a pandas DataFrame

!pip install beautifulsoup4
from bs4 import BeautifulSoup # import BeautifulSoup library

# Matplotlib and associated plotting modules
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as colors

# install wordcloud
!conda install -c conda-forge wordcloud==1.8.1 --yes

# import wordcloud and its set of stopwords
from wordcloud import WordCloud, STOPWORDS

# import k-means from clustering stage
from sklearn.cluster import KMeans

!conda install -c conda-forge folium=0.5.0 --yes
import folium # map rendering library

print('Libraries installed and imported.')

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

Requirement already satisfied: beautifulsoup4 in c:\users\jorge\anaconda3\lib\site-packages (4.8.2)
Requirement already satisfied: soupsieve>=1.2 in c:\users\jorge\anaconda3\lib\site-packages (from beautifulsoup4) (1.9.5)
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

Libraries installed and imported.

3.4. Scraping the list of postal codes in London city dataset from Milesfaster website

The raw data is extracted from Milesfaster website, then processed parsing the html document and finally converted into a DataFrame.

# reading the Milesfaster website content of the server's response
url='https://www.milesfaster.co.uk/london-postcodes-list.htm'
response = requests.get(url)

# parsing the html document from the Milesfaster website using BeautifulSoup library
data = response.text
London_data = BeautifulSoup(data, 'html.parser')
milesfaster_table = London_data.find('table')

# converting the Milesfaster html table into a DataFrame using Pandas library
df_PCLondon = pd.read_html(str(milesfaster_table))[0]

# displaying the first 10 results in the df_PCLondon DataFrame
df_PCLondon.head(10)

As can be observed in the first 10 results of the DataFrame df_PCLondon, this DataFrame should be splitted because the postal codes are displayed in two columns simultaneously. The df_PCLondon DataFrame should be splitted into 2 separated DataFrames.

# This new DataFrame df1 gets the first 2 columns of df_PCLondon DataFrame
df1 = df_PCLondon[[0, 1]]

# displaying the first 10 results in the df1 DataFrame
df1.head(10)

# This new DataFrame df2 gets the last 2 columns of df_PCLondon DataFrame
df2 = df_PCLondon[[2, 3]]

# displaying the first 10 results in the df2 DataFrame
df2.head(10)

As may be seen in DataFrames df1 and df2, columns labels don't match. By renaming columns labels in DataFrame df2 with the same labels of DataFrame df1 both DataFrames can be merged in a single DataFrame.

# renaming columns labels in df2 DataFrame
df2 = df2.rename(columns={2: 0, 3: 1})

# displaying the first 10 results in the df2 DataFrame
df2.head(10)

Now DataFrames df1 and df2 can be merged in a single DataFrame.

# This code merges DataFrames df1 and df2 in a single DataFrame df_PCLondon
df_PCLondon = df1.append(df2)

# printing the complete DataFrame df_PCLondon
print(df_PCLondon)

       0                                           1
0     E1              Whitechapel, Stepney, Mile End
1    E1W                                     Wapping
2     E2                   Bethnal Green, Shoreditch
3     E3                         Bow, Bromley-by-Bow
4     E4                     Chingford, Highams Park
5     E5                                     Clapton
6     E6                                    East Ham
7     E7                     Forest Gate, Upton Park
8     E8                            Hackney, Dalston
9     E9                           Hackney, Homerton
10   E10                                      Leyton
11   E11                                 Leytonstone
12   E12                                  Manor Park
13   E13                                    Plaistow
14   E14   Poplar, Millwall, Isle of Dogs, Docklands
15   E15                         Stratford, West Ham
16   E16     Canning Town, North Woolwich, Docklands
17   E17                                 Walthamstow
18   E18                              South Woodford
19   E20                                Olympic Park
20   NaN                                         NaN
21   WC1                       Bloomsbury, Grays Inn
22   WC2              Covent Garden, Holborn, Strand
23   NaN                                         NaN
24   EC1             Clerkenwell, Finsbury, Barbican
25   EC2                  Moorgate, Liverpool Street
26   EC3               Monument, Tower Hill, Aldgate
27   EC4                     Fleet Street, St. Pauls
28   NaN                                         NaN
29    N1             Islington, Barnsbury, Canonbury
30    N2                               East Finchley
31    N3                            Finchley Central
32    N4                  Finsbury Park, Manor House
33    N5                                    Highbury
34    N6                                    Highgate
35    N7                                    Holloway
36    N8                         Hornsey, Crouch End
37    N9                              Lower Edmonton
38   N10                                Muswell Hill
39   N11                Friern Barnet, New Southgate
40   N12               North Finchley, Woodside Park
41   N13                               Palmers Green
42   N14                                   Southgate
43   N15                               Seven Sisters
44   N16              Stoke Newington, Stamford Hill
45   N17                                   Tottenham
46   N18                              Upper Edmonton
47   N19                       Archway, Tufnell Park
48   N20                       Whetstone, Totteridge
49   N21                              Winchmore Hill
50   N22                Wood Green, Alexandra Palace
51   NaN                                         NaN
52   NW1                   Regents Park, Camden Town
53   NW2                        Cricklewood, Neasden
54   NW3                    Hampstead, Swiss Cottage
55   NW4                         Hendon, Brent Cross
56   NW5                                Kentish Town
57   NW6        West Hampstead, Kilburn, Queens Park
58   NW7                                   Mill Hill
59   NW8                               St Johns Wood
60   NW9                         Kinsbury, Colindale
61  NW10          Willesden, Harlesden, Kensal Green
62  NW11         Golders Green, Hampstead Gdn Suburb
63   NaN                                         NaN
0    SE1    Waterloo, Bermondsey, Southwark, Borough
1    SE2                                  Abbey Wood
2    SE3                  Blackheath, Westcombe Park
3    SE4      Brockley, Crofton Park, Honor Oak Park
4    SE5                                  Camberwell
5    SE6           Catford, Hither Green, Bellingham
6    SE7                                    Charlton
7    SE8                                    Deptford
8    SE9                          Eltham, Mottingham
9   SE10                                   Greenwich
10  SE11                                     Lambeth
11  SE12                             Lee, Grove Park
12  SE13                      Lewisham, Hither Green
13  SE14                   New Cross, New Cross Gate
14  SE15                            Peckham, Nunhead
15  SE16  Rotherhithe, South Bermonsey, Surrey Docks
16  SE17                 Walworth, Elephant & Castle
17  SE18                         Woolwich, Plumstead
18  SE19               Upper Norwood, Crystal Palace
19  SE20                              Penge, Anerley
20  SE21                                     Dulwich
21  SE22                                East Dulwich
22  SE23                                 Forest Hill
23  SE24                                  Herne Hill
24  SE25                               South Norwood
25  SE26                                    Sydenham
26  SE27                    West Norwood, Tulse Hill
27  SE28                                  Thamesmead
28   NaN                                         NaN
29   SW1             Westminster, Belgravia, Pimlico
30   SW2                     Brixton, Streatham Hill
31   SW3                           Chelsea, Brompton
32   SW4                                     Clapham
33   SW5                                 Earls Court
34   SW6                       Fulham, Parsons Green
35   SW7                            South Kensington
36   SW8                    South Lambeth, Nine Elms
37   SW9                          Stockwell, Brixton
38  SW10                   West Brompton, Worlds End
39  SW11                 Battersea, Clapham Junction
40  SW12                                      Balham
41  SW13                           Barnes, Castelnau
42  SW14                        Mortlake, East Sheen
43  SW15                          Putney, Roehampton
44  SW16                          Streatham, Norbury
45  SW17                                     Tooting
46  SW18                      Wandsworth, Earlsfield
47  SW19                           Wimbledon, Merton
48  SW20                South Wimbledon, Raynes Park
49   NaN                                         NaN
50    W1                   Mayfair, Marylebone, Soho
51    W2                       Bayswater, Paddington
52    W3                                       Acton
53    W4                                    Chiswick
54    W5                                      Ealing
55    W6                                 Hammersmith
56    W7                                     Hanwell
57    W8                                  Kensington
58    W9                  Maida Vale, Warwick Avenue
59   W10            Ladbroke Grove, North Kensington
60   W11                  Notting Hill, Holland Park
61   W12                              Shepherds Bush
62   W13                                 West Ealing
63   W14                             West Kensington

3.5. Data Cleaning

As can be noticed in the resulting df_PCLondon DataFrame the columns labels should be renamed to a more descriptive labels. There are NaN values in some rows that must be removed as well.

# renaming the columns names
df_PCLondon.columns = ['Postal Code', 'District']

# displaying the first 10 results in the df_PCLondon DataFrame
df_PCLondon.head(10)

# removing rows with NaN values
df_PCLondon=df_PCLondon[df_PCLondon['Postal Code'].str.contains("NaN") == False].reset_index()

# printing the df_PCLondon DataFrame
print(df_PCLondon)

     index Postal Code                                    District
0        0          E1              Whitechapel, Stepney, Mile End
1        1         E1W                                     Wapping
2        2          E2                   Bethnal Green, Shoreditch
3        3          E3                         Bow, Bromley-by-Bow
4        4          E4                     Chingford, Highams Park
5        5          E5                                     Clapton
6        6          E6                                    East Ham
7        7          E7                     Forest Gate, Upton Park
8        8          E8                            Hackney, Dalston
9        9          E9                           Hackney, Homerton
10      10         E10                                      Leyton
11      11         E11                                 Leytonstone
12      12         E12                                  Manor Park
13      13         E13                                    Plaistow
14      14         E14   Poplar, Millwall, Isle of Dogs, Docklands
15      15         E15                         Stratford, West Ham
16      16         E16     Canning Town, North Woolwich, Docklands
17      17         E17                                 Walthamstow
18      18         E18                              South Woodford
19      19         E20                                Olympic Park
20      21         WC1                       Bloomsbury, Grays Inn
21      22         WC2              Covent Garden, Holborn, Strand
22      24         EC1             Clerkenwell, Finsbury, Barbican
23      25         EC2                  Moorgate, Liverpool Street
24      26         EC3               Monument, Tower Hill, Aldgate
25      27         EC4                     Fleet Street, St. Pauls
26      29          N1             Islington, Barnsbury, Canonbury
27      30          N2                               East Finchley
28      31          N3                            Finchley Central
29      32          N4                  Finsbury Park, Manor House
30      33          N5                                    Highbury
31      34          N6                                    Highgate
32      35          N7                                    Holloway
33      36          N8                         Hornsey, Crouch End
34      37          N9                              Lower Edmonton
35      38         N10                                Muswell Hill
36      39         N11                Friern Barnet, New Southgate
37      40         N12               North Finchley, Woodside Park
38      41         N13                               Palmers Green
39      42         N14                                   Southgate
40      43         N15                               Seven Sisters
41      44         N16              Stoke Newington, Stamford Hill
42      45         N17                                   Tottenham
43      46         N18                              Upper Edmonton
44      47         N19                       Archway, Tufnell Park
45      48         N20                       Whetstone, Totteridge
46      49         N21                              Winchmore Hill
47      50         N22                Wood Green, Alexandra Palace
48      52         NW1                   Regents Park, Camden Town
49      53         NW2                        Cricklewood, Neasden
50      54         NW3                    Hampstead, Swiss Cottage
51      55         NW4                         Hendon, Brent Cross
52      56         NW5                                Kentish Town
53      57         NW6        West Hampstead, Kilburn, Queens Park
54      58         NW7                                   Mill Hill
55      59         NW8                               St Johns Wood
56      60         NW9                         Kinsbury, Colindale
57      61        NW10          Willesden, Harlesden, Kensal Green
58      62        NW11         Golders Green, Hampstead Gdn Suburb
59       0         SE1    Waterloo, Bermondsey, Southwark, Borough
60       1         SE2                                  Abbey Wood
61       2         SE3                  Blackheath, Westcombe Park
62       3         SE4      Brockley, Crofton Park, Honor Oak Park
63       4         SE5                                  Camberwell
64       5         SE6           Catford, Hither Green, Bellingham
65       6         SE7                                    Charlton
66       7         SE8                                    Deptford
67       8         SE9                          Eltham, Mottingham
68       9        SE10                                   Greenwich
69      10        SE11                                     Lambeth
70      11        SE12                             Lee, Grove Park
71      12        SE13                      Lewisham, Hither Green
72      13        SE14                   New Cross, New Cross Gate
73      14        SE15                            Peckham, Nunhead
74      15        SE16  Rotherhithe, South Bermonsey, Surrey Docks
75      16        SE17                 Walworth, Elephant & Castle
76      17        SE18                         Woolwich, Plumstead
77      18        SE19               Upper Norwood, Crystal Palace
78      19        SE20                              Penge, Anerley
79      20        SE21                                     Dulwich
80      21        SE22                                East Dulwich
81      22        SE23                                 Forest Hill
82      23        SE24                                  Herne Hill
83      24        SE25                               South Norwood
84      25        SE26                                    Sydenham
85      26        SE27                    West Norwood, Tulse Hill
86      27        SE28                                  Thamesmead
87      29         SW1             Westminster, Belgravia, Pimlico
88      30         SW2                     Brixton, Streatham Hill
89      31         SW3                           Chelsea, Brompton
90      32         SW4                                     Clapham
91      33         SW5                                 Earls Court
92      34         SW6                       Fulham, Parsons Green
93      35         SW7                            South Kensington
94      36         SW8                    South Lambeth, Nine Elms
95      37         SW9                          Stockwell, Brixton
96      38        SW10                   West Brompton, Worlds End
97      39        SW11                 Battersea, Clapham Junction
98      40        SW12                                      Balham
99      41        SW13                           Barnes, Castelnau
100     42        SW14                        Mortlake, East Sheen
101     43        SW15                          Putney, Roehampton
102     44        SW16                          Streatham, Norbury
103     45        SW17                                     Tooting
104     46        SW18                      Wandsworth, Earlsfield
105     47        SW19                           Wimbledon, Merton
106     48        SW20                South Wimbledon, Raynes Park
107     50          W1                   Mayfair, Marylebone, Soho
108     51          W2                       Bayswater, Paddington
109     52          W3                                       Acton
110     53          W4                                    Chiswick
111     54          W5                                      Ealing
112     55          W6                                 Hammersmith
113     56          W7                                     Hanwell
114     57          W8                                  Kensington
115     58          W9                  Maida Vale, Warwick Avenue
116     59         W10            Ladbroke Grove, North Kensington
117     60         W11                  Notting Hill, Holland Park
118     61         W12                              Shepherds Bush
119     62         W13                                 West Ealing
120     63         W14                             West Kensington

With the purpose of working with df_PCLondon DataFrame onwards, the index column must be removed.

# dropping the first column "index"
df_PCLondon.drop(['index'], axis = 1, inplace = True)

# displaying the first 10 results in the cleaned DataFrame
df_PCLondon.head(10)

Let's get the size of the df_PCLondon DataFrame.

# printing the size of the df_PCLondon DataFrame
print('The shape for the df_PCLondon DataFrame is:', df_PCLondon.shape)

The shape for the df_PCLondon DataFrame is: (121, 2)

The DataFrame contains 121 Postal Codes (rows) and 2 columns: Postal Code and Districts.

3.6. Adding the latitude and longitude coordinates to the DataFrame

A new function will be created to get the coordinates for all districts using ArcGIS, a powerful library useful for spatial analysis and mapping.

# creating a function to get the coordinates (latitude, longitude) for all districts
def get_geocode(arcgis_geocoder):
    
    # Initialize the location (latitude and longitude) to None
    lat_lng_coords = None
    
    # loop until the coordinates are obtained
    while(lat_lng_coords is None):
        g = geocoder.arcgis('{}, London, United Kingdom'.format(arcgis_geocoder))
        lat_lng_coords = g.latlng
    return lat_lng_coords

Let's test the function get_geocode created previously with two random Postal Codes to get their latitude and longitude coordinates.

# sampling two random chosen Postal codes to get their latitude and longitude coordinates using the function get_geocode
sample1 = get_geocode('NW11')
print(sample1)

sample2 = get_geocode('SW9')
print(sample2)

[51.576730000000055, -0.19694999999995844]
[51.47001000000006, -0.11248999999997977]

As illustrated the function get_geocode is working well. The function will be applied to df_PCLondon DataFrame.

# storing the location data (latitude and longitude) in the DataFrame df_PCLondon
postal_codes = df_PCLondon['Postal Code']
coordinates = [get_geocode(postal_codes) for postal_codes in postal_codes.tolist()]

# The coordinates are joined to df_PCLondon DataFrame to create a new DataFrame with the latitude and longitude
df_PCLondon_coordinates = pd.DataFrame(coordinates, columns = ['Latitude', 'Longitude'])
df_PCLondon['Latitude'] = df_PCLondon_coordinates['Latitude']
df_PCLondon['Longitude'] = df_PCLondon_coordinates['Longitude']

# displaying the first 10 results in the new merged DataFrame df_PCLondon
df_PCLondon.head(10)

The new DataFrame will be used to generate the venues for each district using Foursquare API.

3.7. Using Foursquare API to explore the districts and segment them

The geographical coordinates of London, U.K. will be obtained using Nominatim.

# getting the geographical coordinates of London city
address = 'London, United Kingdom'

geolocator = Nominatim(user_agent = "ln_explorer")
location = geolocator.geocode(address)
latitude = location.latitude
longitude = location.longitude
print('The geographical coordinates of London are: {}, {}'.format(latitude, longitude))

The geographical coordinates of London are: 51.5073219, -0.1276474

The districts of London city will be visualized using Folium library.

# creating a map of London city using latitude and longitude values
map_london = folium.Map(location=[latitude, longitude], zoom_start = 11)

# adding markers to London city map
for lat, lng, postal_code, district in zip(df_PCLondon['Latitude'], df_PCLondon['Longitude'], df_PCLondon['Postal Code'], df_PCLondon['District']):
    label = '{}, {}'.format(postal_code, district)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
    [lat, lng],
    radius=5,
    popup=label,
    color='blue',
    fill=True,
    fill_color='#3186cc',
    fill_opacity=0.7,
    ).add_to(map_london)
    
map_london

# defining Foursquare credentials and version
CLIENT_ID = 'SF3QURMDBEA11X4EDFJPY30R5AYACPJBBHL0I1EAVPYLGJC0' # Foursquare ID
CLIENT_SECRET = 'O5PEG4DFUJKZW3RNE4E2WSSJHUSVCIWPF3BQUNQJHIHZLUNJ' # Foursquare Secret
VERSION = '20180605' # Foursquare API version
LIMIT = 100 # A default Foursquare API limit value

print('Your credentials:')
print('CLIENT_ID: ' + CLIENT_ID)
print('CLIENT_SECRET' + CLIENT_SECRET)

Your credentials:
CLIENT_ID: SF3QURMDBEA11X4EDFJPY30R5AYACPJBBHL0I1EAVPYLGJC0
CLIENT_SECRETO5PEG4DFUJKZW3RNE4E2WSSJHUSVCIWPF3BQUNQJHIHZLUNJ

Let's create a function to repeat the same process to all districts in London city.

# exploring the districts in London city
def getNearbyVenues(names, latitudes, longitudes, radius=2000):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        print(name)
        
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID,
            CLIENT_SECRET,
            VERSION,
            lat,
            lng,
            radius,
            LIMIT)
        
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name,
            lat,
            lng,
            v['venue']['name'],
            v['venue']['location']['lat'],
            v['venue']['location']['lng'],
            v['venue']['categories'][0]['name']) for v in results])
        
    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['District',
                            'District Latitude',
                            'District Longitude',
                            'Venue',
                            'Venue Latitude',
                            'Venue Longitude',
                            'Venue Category']
    
    return(nearby_venues)

# returning a list of venues near the current location
london_venues = getNearbyVenues(names=df_PCLondon['District'],
                                latitudes=df_PCLondon['Latitude'],
                                longitudes=df_PCLondon['Longitude'])

Whitechapel, Stepney, Mile End
Wapping
Bethnal Green, Shoreditch
Bow, Bromley-by-Bow
Chingford, Highams Park
Clapton
East Ham
Forest Gate, Upton Park
Hackney, Dalston
Hackney, Homerton
Leyton
Leytonstone
Manor Park
Plaistow
Poplar, Millwall, Isle of Dogs, Docklands
Stratford, West Ham
Canning Town, North Woolwich, Docklands
Walthamstow
South Woodford
Olympic Park
Bloomsbury, Grays Inn
Covent Garden, Holborn, Strand
Clerkenwell, Finsbury, Barbican
Moorgate, Liverpool Street
Monument, Tower Hill, Aldgate
Fleet Street, St. Pauls
Islington, Barnsbury, Canonbury
East Finchley
Finchley Central
Finsbury Park, Manor House
Highbury
Highgate
Holloway
Hornsey, Crouch End
Lower Edmonton
Muswell Hill
Friern Barnet, New Southgate
North Finchley, Woodside Park
Palmers Green
Southgate
Seven Sisters
Stoke Newington, Stamford Hill
Tottenham
Upper Edmonton
Archway, Tufnell Park
Whetstone, Totteridge
Winchmore Hill
Wood Green, Alexandra Palace
Regents Park, Camden Town
Cricklewood, Neasden
Hampstead, Swiss Cottage
Hendon, Brent Cross
Kentish Town
West Hampstead, Kilburn, Queens Park
Mill Hill
St Johns Wood
Kinsbury, Colindale
Willesden, Harlesden, Kensal Green
Golders Green, Hampstead Gdn Suburb
Waterloo, Bermondsey, Southwark, Borough
Abbey Wood
Blackheath, Westcombe Park
Brockley, Crofton Park, Honor Oak Park
Camberwell
Catford, Hither Green, Bellingham
Charlton
Deptford
Eltham, Mottingham
Greenwich
Lambeth
Lee, Grove Park
Lewisham, Hither Green
New Cross, New Cross Gate
Peckham, Nunhead
Rotherhithe, South Bermonsey, Surrey Docks
Walworth, Elephant & Castle
Woolwich, Plumstead
Upper Norwood, Crystal Palace
Penge, Anerley
Dulwich
East Dulwich
Forest Hill
Herne Hill
South Norwood
Sydenham
West Norwood, Tulse Hill
Thamesmead
Westminster, Belgravia, Pimlico
Brixton, Streatham Hill
Chelsea, Brompton
Clapham
Earls Court
Fulham, Parsons Green
South Kensington
South Lambeth, Nine Elms
Stockwell, Brixton
West Brompton, Worlds End
Battersea, Clapham Junction
Balham
Barnes, Castelnau
Mortlake, East Sheen
Putney, Roehampton
Streatham, Norbury
Tooting
Wandsworth, Earlsfield
Wimbledon, Merton
South Wimbledon, Raynes Park
Mayfair, Marylebone, Soho
Bayswater, Paddington
Acton
Chiswick
Ealing
Hammersmith
Hanwell
Kensington
Maida Vale, Warwick Avenue
Ladbroke Grove, North Kensington
Notting Hill, Holland Park
Shepherds Bush
West Ealing
West Kensington

Let's check the size of the resulting DataFrame.

# printing the first 10 nearby venues in London city districts
london_venues.head(10)

# size of the london_venues DataFrame
print('The shape of london_venues DataFrame is:', london_venues.shape)

The shape of london_venues DataFrame is: (11145, 7)

# checking the number of venues in each district
london_venues.groupby('District').count()

4. Methodology

The methodology applied in this project is as follows:

1. Foursquare API will be used to explore all venues of all districts and analyze each district in London city.

1. The most relevant venues to visit will be retrieved from the entire DataFrame and visualized in bar plots.

1. The most common venues will be analyzed using word cloud to look for the most common words and translate them into venues.

1. Each district will be grouped with the 10 most common venues.

1. All districts in London city will be clustered, visualized and examined using k-means algorithm.

1. The resulting clusters will be analyzed using word cloud to validate the analyses performed in previous sections.

5. Explanatory Analysis

This section consists in performing an explanatory data analysis and deriving some additional information from the raw data. The first step is analyzing each distric looking for venues.

5.1. Analyzing districts

# one hot encoding
london_onehot = pd.get_dummies(london_venues[['Venue Category']], prefix="", prefix_sep="")

# adding district column back to DataFrame
london_onehot['District'] = london_venues['District']

# moving district column to the first column
fixed_columns = [london_onehot.columns[-1]] + list(london_onehot.columns[:-1])
london_onehot = london_onehot[fixed_columns]

london_onehot.head()

Let's examine the new DataFrame size.

# getting the size of the london_onehot DataFrame
london_onehot.shape

(11145, 348)

The most relevant venue categories will be counted with the purpose of evaluating whether or not London is a good place to travel.

# getting the number of Restaurants in London City
print('The number of Restaurants in London city is:', (london_onehot['Restaurant'].values==1).sum())

# getting the number of Train Stations in London City
print('The number of Train Stations in London city is:', (london_onehot['Train Station'].values==1).sum())

# getting the number of Aquariums in London City
print('The number of Aquariums in London city is:', (london_onehot['Aquarium'].values==1).sum())

# getting the number of Art Museums in London City
print('The number of Art Museums in London city is:', (london_onehot['Art Museum'].values==1).sum())

# getting the number of Bookstores in London City
print('The number of Bookstores in London city is:', (london_onehot['Bookstore'].values==1).sum())

# getting the number of Gyms in London City
print('The number of Gyms in London city is:', (london_onehot['Gym'].values==1).sum())

# getting the number of Hotels in London City
print('The number of Hotels in London city is:', (london_onehot['Hotel'].values==1).sum())

# getting the number of Bars in London City
print('The number of Bars in London city is:', (london_onehot['Bar'].values==1).sum())

# getting the number of Shopping Malls in London City
print('The number of Shopping Malls in London city is:', (london_onehot['Shopping Mall'].values==1).sum())

# getting the number of Gift Shops in London City
print('The number of Gift Shops in London city is:', (london_onehot['Gift Shop'].values==1).sum())

# getting the number of Theaters in London City
print('The number of Theaters in London city is:', (london_onehot['Theater'].values==1).sum())

# getting the number of Zoos in London City
print('The number of Zoos in London city is:', (london_onehot['Zoo'].values==1).sum())

The number of Restaurants in London city is: 163
The number of Train Stations in London city is: 57
The number of Aquariums in London city is: 4
The number of Art Museums in London city is: 20
The number of Bookstores in London city is: 85
The number of Gyms in London city is: 59
The number of Hotels in London city is: 254
The number of Bars in London city is: 159
The number of Shopping Malls in London city is: 20
The number of Gift Shops in London city is: 26
The number of Theaters in London city is: 89
The number of Zoos in London city is: 4

Based on this information retrieved from london_onehot DataFrame containing the number of restaurants, train stations, aquariums, art museums, bookstores, gyms, hotels, bars, shopping malls, gift shops, theatres and zoos in London city, a new DataFrame will be created to visualize these venues in a bar plot.

# building a new DataFrame containing the venues previously explored
data1 ={'Venue Type': ['Restaurants', 'Train Stations', 'Aquariums', 'Art Museums', 'Bookstores', 
                 'Gyms', 'Hotels', 'Bars', 'Shopping Malls', 'Gift Shops', 'Theaters', 'Zoos'],
      'Total Venues': [163, 57, 4, 20, 83, 53, 243, 160, 20, 23, 92, 5]}

df_london_num_venues = pd.DataFrame(data1, columns = ['Venue Type', 'Total Venues'])

print(df_london_num_venues)

        Venue Type  Total Venues
0      Restaurants           163
1   Train Stations            57
2        Aquariums             4
3      Art Museums            20
4       Bookstores            83
5             Gyms            53
6           Hotels           243
7             Bars           160
8   Shopping Malls            20
9       Gift Shops            23
10        Theaters            92
11            Zoos             5

Let's build the bar plot with the most relevant venues in London city.

# magic function that renders the figure in a notebook
%matplotlib inline 

# bar plot of the 'df_london_num_venues' DataFrame
plt.figure(figsize=(12, 8))

df3 = df_london_num_venues.sort_values('Total Venues', ascending=False)
plt.bar(df3['Venue Type'], df3['Total Venues'])

# title
plt.title('Most relevant venues in London city', fontsize=16)

# labels
plt.xlabel('Venue Type', fontsize=14)
plt.ylabel('Total Venues', fontsize=14)

# rotating x label
plt.xticks(rotation=60)

# increasing the default intervals for y-axis
plt.yticks(np.arange(0, 270, 20))

# displaying the bar plot
plt.show()

As can be noted from the bar plot above, from the venues selected, hotels are the most common venue in London city, followed by restaurants. Now hotels will be grouped by district to find which districts have most hotels in London city.

# grouping london_onehot DataFrame by district and getting the Hotel column with the total of hotels by district
df_london_hotels = london_onehot.groupby('District', axis=0)['Hotel'].sum().reset_index()

# printing the first 10 results
print(df_london_hotels.head(10))

                      District  Hotel
0                   Abbey Wood      0
1                        Acton      5
2        Archway, Tufnell Park      0
3                       Balham      0
4            Barnes, Castelnau      0
5  Battersea, Clapham Junction      1
6        Bayswater, Paddington      6
7    Bethnal Green, Shoreditch      2
8   Blackheath, Westcombe Park      1
9        Bloomsbury, Grays Inn      4

Let's build the bar plot containing the 10 districts with the largest number of hotels.

# magic function that renders the figure in a notebook
%matplotlib inline 

# bar plot of the 'df_london_hotels' DataFrame
plt.figure(figsize=(12, 8))

df4 = df_london_hotels.sort_values('Hotel', ascending=False).head(10)
plt.bar(df4['District'], df4['Hotel'])

# title
plt.title('Top 10 districts by hotel availability', fontsize=16)

# labels
plt.xlabel('District', fontsize=14)
plt.ylabel('Total of Hotels', fontsize=14)

# rotating x label
plt.xticks(rotation=90)

# increasing the default intervals for y-axis
plt.yticks(np.arange(0, 17, 1))

# displaying the bar plot
plt.show()

As previously mentioned, restaurants is the next category from the venues selected. The next analysis consists in grouping restaurants by district to find which districts have most restaurants in London city. National and international restaurants will be counted to get an idea about the variety of cuisines available in London city restaurants.

# getting the number of African Restaurants in London City
print('The number of African Restaurants in London city is:', (london_onehot['African Restaurant'].values==1).sum())

# getting the number of American Restaurants in London City
print('The number of American Restaurants in London city is:', (london_onehot['American Restaurant'].values==1).sum())

# getting the number of Asian Restaurants in London City
print('The number of Asian Restaurants in London city is:', (london_onehot['Asian Restaurant'].values==1).sum())

# getting the number of Chinese Restaurants in London City
print('The number of Chinese Restaurants in London city is:', (london_onehot['Chinese Restaurant'].values==1).sum())

# getting the number of English Restaurants in London City
print('The number of English Restaurants in London city is:', (london_onehot['English Restaurant'].values==1).sum())

# getting the number of Indian Restaurants in London City
print('The number of Indian Restaurants in London city is:', (london_onehot['Indian Restaurant'].values==1).sum())

# getting the number of Italianh Restaurants in London City
print('The number of Italian Restaurants in London city is:', (london_onehot['Italian Restaurant'].values==1).sum())

# getting the number of Japanese Restaurants in London City
print('The number of Japanese Restaurants in London city is:', (london_onehot['Japanese Restaurant'].values==1).sum())

# getting the number of Spanish Restaurants in London City
print('The number of Spanish Restaurants in London city is:', (london_onehot['Spanish Restaurant'].values==1).sum())

# getting the number of Turkish Restaurants in London City
print('The number of Turkish Restaurants in London city is:', (london_onehot['Turkish Restaurant'].values==1).sum())

The number of African Restaurants in London city is: 22
The number of American Restaurants in London city is: 14
The number of Asian Restaurants in London city is: 49
The number of Chinese Restaurants in London city is: 33
The number of English Restaurants in London city is: 42
The number of Indian Restaurants in London city is: 170
The number of Italian Restaurants in London city is: 271
The number of Japanese Restaurants in London city is: 88
The number of Spanish Restaurants in London city is: 18
The number of Turkish Restaurants in London city is: 143

A new DataFrame based on this information retrieved from london_onehot DataFrame will be created containing the number of restaurants by origin or cuisine type in London city with the purpose of visualizing these restaurants in a bar plot.

# creating a new DataFrame containing the different restaurants by cuisine type in London city
data2 ={'Restaurant Type': ['African', 'American', 'Asian', 'Chinese', 'English', 'Indian', 'Italian', 'Japanese',
                          'Spanish', 'Turkish'],
      'Total of Restaurants': [23, 14, 44, 31, 44, 167, 280, 86, 18, 142]}

df_london_restaurant_types = pd.DataFrame(data2, columns = ['Restaurant Type', 'Total of Restaurants'])

print(df_london_restaurant_types)

  Restaurant Type  Total of Restaurants
0         African                    23
1        American                    14
2           Asian                    44
3         Chinese                    31
4         English                    44
5          Indian                   167
6         Italian                   280
7        Japanese                    86
8         Spanish                    18
9         Turkish                   142

Let's build the bar plot with the total of different types of restaurants in London city.

# magic function that renders the figure in a notebook
%matplotlib inline 

# bar plot of the 'df_london_restaurant_types' DataFrame
plt.figure(figsize=(12, 8))

df6 = df_london_restaurant_types.sort_values('Total of Restaurants', ascending=False)
plt.bar(df6['Restaurant Type'], df6['Total of Restaurants'])

# title
plt.title('Restaurants in London city by cuisine type', fontsize=16)

# labels
plt.xlabel('Restaurant Type', fontsize=14)
plt.ylabel('Total of Restaurants', fontsize=14)

# increasing the default intervals for y-axis
plt.yticks(np.arange(0, 300, 20))

# displaying the bar plot
plt.show()

As observed above, Italian and Indian restaurants lead the cuisines in London city. A more detailed analysis will reveal the spatial distribution of such restaurants by district.

# grouping london_onehot DataFrame by district and getting the Italian Restaurants column with the number of these restaurants by district
df_london_italian_restaurants = london_onehot.groupby('District', axis=0)['Italian Restaurant'].sum().reset_index()

# printing the first 10 results
print(df_london_italian_restaurants.head(10))

                      District  Italian Restaurant
0                   Abbey Wood                   0
1                        Acton                   2
2        Archway, Tufnell Park                   5
3                       Balham                   5
4            Barnes, Castelnau                   3
5  Battersea, Clapham Junction                   5
6        Bayswater, Paddington                   2
7    Bethnal Green, Shoreditch                   3
8   Blackheath, Westcombe Park                   3
9        Bloomsbury, Grays Inn                   0

Let's build the bar plot containing the 10 districts with the largest number of italian restaurants.

# magic function that renders the figure in a notebook
%matplotlib inline 

# bar plot of the 'df_london_italian_restaurants' DataFrame
plt.figure(figsize=(12, 8))

df7 = df_london_italian_restaurants.sort_values('Italian Restaurant', ascending=False).head(10)
plt.bar(df7['District'], df7['Italian Restaurant'])

# title
plt.title('Top 10 districts by Italian Restaurants', fontsize=16)

# labels
plt.xlabel('District', fontsize=14)
plt.ylabel('Total of Italian Restaurants', fontsize=14)

# rotating x label
plt.xticks(rotation=90)

# increasing the default intervals for y-axis
plt.yticks(np.arange(0, 8, 1))

# displaying the bar plot
plt.show()

# grouping london_onehot DataFrame by district and getting the Indian Restaurants column with the number of these restaurants by district
df_london_indian_restaurants = london_onehot.groupby('District', axis=0)['Indian Restaurant'].sum().reset_index()

# printing the first 10 results
print(df_london_indian_restaurants.head(10))

                      District  Indian Restaurant
0                   Abbey Wood                  0
1                        Acton                  1
2        Archway, Tufnell Park                  2
3                       Balham                  1
4            Barnes, Castelnau                  1
5  Battersea, Clapham Junction                  1
6        Bayswater, Paddington                  1
7    Bethnal Green, Shoreditch                  2
8   Blackheath, Westcombe Park                  1
9        Bloomsbury, Grays Inn                  1

Let's build the bar plot containing the 10 districts with the largest number of indian restaurants.

# magic function that renders the figure in a notebook
%matplotlib inline 

# bar plot of the 'df_london_indian_restaurants' DataFrame
plt.figure(figsize=(12, 8))

df8 = df_london_indian_restaurants.sort_values('Indian Restaurant', ascending=False).head(10)
plt.bar(df8['District'], df8['Indian Restaurant'])

# title
plt.title('Top 10 districts by Indian Restaurants', fontsize=16)

# labels
plt.xlabel('District', fontsize=14)
plt.ylabel('Total of Indian Restaurants', fontsize=14)

# rotating x label
plt.xticks(rotation=90)

# increasing the default intervals for y-axis
plt.yticks(np.arange(0, 9, 1))

# displaying the bar plot
plt.show()

The next step consists in grouping rows by district and taking the mean of the frequency of occurrence of each category.

# grouping rows by district
london_grouped = london_onehot.groupby('District').mean().reset_index()
london_grouped.head()

Let's confirm the new size.

# size of the london_grouped DataFrame
london_grouped.shape

(121, 348)

# grouping the 10 common venues in each district
num_top_venues = 10 # Top common venues needed
for district in london_grouped['District']:
    print("---"+district+"---")
    temp = london_grouped[london_grouped['District'] == district].T.reset_index()
    temp.columns = ['venue', 'freq']
    temp = temp.iloc[1:]
    temp['freq'] = temp['freq'].astype(float)
    temp = temp.round({'freq': 2})
    print(temp.sort_values('freq', ascending = False).reset_index(drop = True).head(num_top_venues))
    print('\n')